Performance Assessment and Incremental Evaluation of Rule-based Systems

Rule-based systems that are easily testable are required for high reliability applications. However, as a rule base evolves, developers prefer incremental evaluation owing to the high cost of regression testing. For, quality aatd reliability improvement, researchers advocate that the evaluation phase be integrated with development: thus, incremental evaluation becomes more important in this context. In this paper, we propose a three-tiered lifecycle model for integrating evaluation in a rulebased system life cycle. We then explore two specific issues from a system’s evaluation perspective: (1) the limitation of functional testing for performance evaluation; and (2) the optimization the cost and effort for evaluation by using incremental methods, whenever possible. This research was motivated by our experience in the design and evaluation of rule-based systems. Introduction and Motivation The design of rule bases is often plagued by various anomalies whose detection requires a variety of evaluation procedures (O’Keefe & Lee 1990; O’Keefe & O’Leary 1993; Kiper 1992; Guida & Mauri 1993). The evaluation processes have been traditionally classified as verification, validation, and performance analysis (or measurement). Verification detects the anomalies due to redundancy, deficiency, circularity and ambivalence; functional validation tests the conformance of a system with its functional requirements; structural validation checks whether the observed functional performance is the result of the correct structure (rule) interactions and tries to capture the system structure in a well defined way; and performance assessment measures the adequacy, optimality, and test case coverage of the system. Not only are these processes non-trivial, but system developers are often unclear where they fit in a system life-cycle (Hamilton, Kelley, & Culbert 1991). One of the primary reasons that makes the evaluation of rule-based systems harder is the lack of a well defined link between system conception and its realization. In fact, in many cases, there does Life-cycle